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Improving  the  Effectiveness  of  Army  Distributed  Learning 

A  Research  and  Policy  Agenda 

Susan  G.  Straus,  Jolene  Galegher,  Michael  G.  Shanley,  Joy  S.  Moini 


As  part  of  an  ambitious  plan  to  transform 
its  training  practices,  the  U.S.  Army  has 
established  a  large  distributed  learning 
(DL)  program.  This  program  is  intended 
to  enable  the  Army  to  provide  decentralized  training 
using  one  or  more  information  and  communication 
technologies  or  a  combination  of  classroom  train¬ 
ing  and  technology-mediated  training.  The  Army 
Distributed  Learning  Program,  or  TADLP,  is  a  com¬ 
prehensive  program  that  is  implementing  DL  through 
digital  training  facilities,  courseware,  learning  man¬ 
agement  systems,  and  other  strategies  and  supporting 
mechanisms.  Under  this  program,  the  Army  is  in  the 
process  of  implementing  a  plan  to  convert  525  train¬ 
ing  courses  to  DL  by  the  year  2010. 1 

DL  in  the  Army  involves  (1)  the  delivery  of  indi¬ 
vidual,  collective,  and  self-development  training  to  sol¬ 
diers  and  units  at  any  time  and  any  place;  (2)  through 
multiple  means  and  technologies;  (3)  using  special 
techniques  for  course  design,  instruction,  methods 
of  communication,  and  organizational  and  admin¬ 
istrative  arrangements  (Moore  and  Kearsley,  1996; 

U.S.  Department  of  the  Army,  Training  Doctrine 
Command  [TRADOC],  2003).  Through  DL,  the 
Army  aims  to  achieve  five  goals.  These  goals  are  to 
(1)  train  to  one  standard,  (2)  improve  unit  readi¬ 
ness,  (3)  reduce  costs,  (4)  improve  morale,  and  (5) 
improve  training  efficiency  (U.S.  Department  of  the 
Army,  TADLP,  2003).  DL  offers  additional  benefits 
by  enabling  self-paced  learning,  opportunities  for 
life-long  learning,  and  automated  tracking  of  student 
performance.  At  the  same  time,  DL  poses  numerous 
challenges  (e.g.,  Young,  1998;  Welsh  et  al.,  2003). 


Many  thanks  to  John  Bondanella  and  Derek  Davison  for  their  contribu¬ 
tions  to  this  research  and  to  Walter  Baer,  James  Crowley,  Colonel  Steven 
Hartman,  Matthew  Lewis,  and  Mr.  Michael  Faughnan  for  their  helpful 
feedback  on  this  report. 

1  The  projected  rate  of  courseware  redesign  was  based  on  an  annual  target 
goal  of  31  courses  per  year  through  FY02  and  47  courses  per  year  there¬ 
after  (U.S.  Department  of  the  Army,  TRADOC,  2001). 


Creating  and  maintaining  DL  courses  is  costly.  It  may 
be  difficult  to  train  some  kinds  of  skills  and  to  estab¬ 
lish  rapport  between  instructors  and  trainees  in  distrib¬ 
uted  settings.  Further,  DL  environments  require  that 
instructors  develop  new  skills  and  modify  their  teach¬ 
ing  strategies.  Students  may  also  need  to  learn  to  use 
new  technologies  and  to  take  a  more  active  approach  to 
learning  than  is  typical  in  a  lecture-type  classroom. 

The  purpose  of  this  report  is  to  examine  the  bene¬ 
fits  and  challenges  of  DL  for  learning.  In  particular, 
we  review  research  that  compares  DL  to  classroom  or 
residential  learning  (RL),  which  is  the  current  standard 
for  training.  We  examine  the  effect  of  DL  on  learning 
and  recommend  strategies  for  the  Army  to  evaluate  and 
enhance  its  own  DL  efforts. 

Previous  reviews  of  learning  in  DL  have  found 
that  this  research  consists  largely  of  anecdotal  reports 
and  studies  that  lack  comparison  groups,  strong  experi¬ 
mental  or  quasi-experimental  designs,  or  grounding 
in  theoretical  principles  (e.g.,  Phipps  and  Merisotis, 
1999;  Wisher  et  al.,  1999). 2  At  the  request  of  Combined 
Armed  Services  Support  Command  of  the  U.S.  Army, 
Straus  et  al.  (2004)  reviewed  empirical  studies  pub¬ 
lished  in  2000-2002  that  compared  DL  and  RL  for 
adult  learners  in  military  and  university  settings.  Our 
review  focused  on  what  students  learn,  or  objective 
learning  outcomes,  as  opposed  to  other  aspects  of 
effectiveness,  such  as  student  satisfaction  or  the  costs 
of  development  or  delivery.3  Our  review  was  restricted 
to  studies  that  used  quantitative  empirical  methods, 
because  this  approach  is  most  likely  to  yield  reliable 
assessments  of  the  relative  effectiveness  of  DL  and 
RL.  We  identified  relevant  studies  by  reviewing  the 
contents  of  27  journals  and  four  sets  of  conference 
proceedings  in  education,  technology,  psychology,  and 


2  The  majority  of  studies  reviewed  in  these  reports  were  published  between 
1986-1998. 

3  Many  studies  of  DL  measure  satisfaction  with  instruction.  Although 
satisfaction  is  a  component  of  training  effectiveness,  it  is  not  a  measure  of 
learning. 
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management  as  well  as  reports  published  by  the  Army 
Research  Institute.  To  supplement  our  analysis,  we 
interviewed  training  developers,  instructors,  and  other 
training  staff  at  five  Army  installations. 

Key  findings  of  this  work  are  as  follows: 

•  Although  hundreds  of  studies  were  examined,  we 
identified  only  13  studies  that  measured  objective 
learning  outcomes  and  had  unambiguous  research 
findings.4 

•  The  course  topics  and  media  used  for  DL  varied 
considerably,  making  it  difficult  to  draw  system¬ 
atic  conclusions  about  instructional  media.  For 
example,  examples  of  media  included  one-way  and 
two-way  video-teletraining  (VTT),  Web-based 
instruction,  computer-based  training,  chat,  and 
self-study.  There  also  were  a  number  of  different 
combinations  of  technologies  used  in  DL  (e.g., 
VTT  alone  or  VTT  plus  chat). 

•  Results  were  consistent  with  other  reviews  that 
found  no  clear  advantage  for  DL  or  RL  on  learning. 

•  Many  of  the  studies  we  identified  were  subject 
to  methodological  limitations,  including  the 
use  of  convenience  samples  rather  than  random 
assignment,  small  sample  sizes,  low  response 
rates  or  differential  response  rates  in  DL  and  RL 
conditions,  and  insufficient  evidence  about  the 
reliability  and  validity  of  measures.  These  short¬ 
comings  are  consistent  with  those  reported  by 
Phipps  and  Merisotis  and  Wisher  et  al.  in  their 
1999  reviews. 

These  findings  of  no  advantage  for  DL  or  RL  were 
inconsistent  with  recent  results  reported  by  Sitzmann 
et  al.  (2005),  who  conducted  a  meta-analysis  of 
studies  that  compared  classroom  instruction  with 
Web-based  instruction  for  adult  learners.  Using 
research  published  from  1996-2005,  they  found  that 
Web-based  instruction  was  superior  to  RL  for  train¬ 
ing  declarative  knowledge  (knowledge  about  facts) 

(77  studies)  but  that  there  was  no  difference  between 
media  for  training  procedural  knowledge  (knowl¬ 
edge  about  processes)  (12  studies).  Flowever,  like  the 
studies  reviewed  by  Straus  et  al.  (2004),  only  a  small 
number  of  studies  (11  studies)  in  the  Sitzmann  et  al. 
review  were  true  experiments  in  which  students  were 
randomly  assigned  to  training  conditions.  Moreover, 


4  We  originally  identified  33  studies  that  appeared  to  fit  our  criteria.  After 
careful  review,  we  eliminated  20  of  them  because  they  measured  outcomes 
other  than  learning  (e.g.,  satisfaction,  graduation  rates),  consisted  of  the 
same  data  published  in  different  journals  or  conference  proceedings, 
or  were  subject  to  methodological  weaknesses  that  rendered  the  results 
ambiguous.  For  instance,  in  several  studies,  differences  in  the  characteris¬ 
tics  of  students  in  the  DL  and  RL  conditions  (e.g.,  in  pre-test  scores,  expe¬ 
rience,  or  age)  were  not  controlled  for  in  the  analyses  of  study  outcomes. 


in  the  experimental  studies,  the  effects  of  medium 
were  reversed,  such  that  students  performed  better 
on  tests  of  declarative  knowledge  in  RL  than  in  DL. 
Thus,  in  the  quasi-experimental  studies  in  Sitzmann 
et  al.  as  well  as  in  other  reviews,  results  showing  no 
differences  between  RL  and  DL  or  better  perfor¬ 
mance  in  DL  may  have  been  due  to  factors  such  as 
selection  biases.  That  is,  these  differences  may  be 
due  to  systematic  differences  in  the  types  of  students 
who  chose  RL  and  DL,  rather  than  to  instructional 
medium.  The  implications  of  these  findings  are  par¬ 
ticularly  important  for  the  Army  to  consider,  given 
that  for  some  courses,  trainees  may  not  have  a  choice 
about  the  instructional  medium. 

Although  studies  such  as  Sitzmann  et  al.  (2005) 
are  promising,  internal  validity  in  most  research  on 
learning  outcomes  in  DL  remains  problematic.  In 
other  words,  the  experimental  designs  used  in  the 
preponderance  of  studies  do  not  allow  conclusions 
about  whether  the  treatment  (DL  or  RL)  caused  the 
observed  effects  on  learning  (see  Cook  and  Campbell, 
1979,  for  a  discussion  of  threats  to  internal  validity). 
This  state  of  affairs  is  of  increasing  concern  because 
technological  capabilities  are  evolving  rapidly,  and 
new  technologies  are  being  deployed  without  an 
empirical  foundation  to  inform  decisions  about  their 
use.  Without  the  knowledge  that  might  be  derived 
from  such  research,  it  is  difficult  to  determine  which 
technologies  are  most  likely  to  be  beneficial  for  par¬ 
ticular  kinds  of  training  and  how  those  technologies 
might  best  be  employed. 

Thus,  we  present  here  three  sets  of  recommenda¬ 
tions  to  help  the  Army  meet  its  goals  for  DL.  The 
first  set  of  recommendations  focuses  on  the  kinds 
of  questions  that  should  be  addressed  in  a  well- 
defined  program  of  research  on  the  effectiveness  of 
DL.  The  second  set  of  recommendations  focuses  on 
improving  the  quality  of  research  by  using  rigorous 
methods  that  yield  credible  results.  Addressing  the 
research  questions  we  propose  and  using  the  research 
techniques  we  describe  in  future  analyses  of  DL  pro¬ 
grams  will  strengthen  the  Army’s  knowledge  base 
and  enable  the  Army  to  increase  training  effective¬ 
ness.  Finally,  we  recommend  a  set  of  policies  that  are 
needed  to  support  the  design,  implementation,  and 
evaluation  of  DL  in  the  Army. 

Establish  a  Research  Agenda  Focused 
on  the  Effects  of  Variations  in  the 
Design  of  Distributed  Learning 

Many  studies  of  DL  have  compared  DL  and  RL, 
but  we  believe  that  focusing  on  DL  alone,  and  the 
factors  that  influence  its  effectiveness,  would  be  more 
valuable.  This  recommendation  is  based  on  two 
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arguments.  First,  it  is  often  impractical  or  too  costly 
to  conduct  rigorous  field  experiments  that  compare 
results  for  the  same  course  given  in  both  RL  and  DL 
formats,  which  means  that  studies  of  this  sort  are 
likely  to  suffer  from  such  methodological  problems 
as  lack  of  random  assignment  and  small  sample  sizes, 
which  undermine  their  value.  Second,  the  Army  is 
already  moving  to  DL,  in  part  because  it  may  offer 
such  advantages  as  the  potential  for  increased  person¬ 
nel  readiness  and  the  ability  to  train  at  any  time  and 
at  any  place.  These  advantages  may  justify  invest¬ 
ments  in  DL  even  if  performance  is  no  better  than  in 
RL.  Thus,  research  focused  on  enhancing  the  quality 
of  DL — rather  than  contrasting  RL  with  DL — is 
more  likely  to  yield  results  that  are  of  practical  value 
to  the  Army. 

Below,  we  identify  specific  questions  for  a  research 
agenda  on  DL.  We  regard  these  questions  as  particu¬ 
larly  important,  but  they  should  be  seen  as  exem¬ 
plars  of  the  kinds  of  questions  that  might  be  asked. 
There  are,  no  doubt,  other  specific  issues  that  might 
be  addressed  in  a  comprehensive  analysis  of  DL. 
Regardless  of  the  particular  issue  addressed,  the  design 
of  specific  DL  configurations,  as  well  as  corresponding 
research  questions  and  measures,  should  be  guided  by 
theory  in  education  and  other  relevant  disciplines. 

Use  a  Student-Centered  Approach  to  Frame 
Research  Questions 

Some  researchers  have  discussed  the  importance  of 
using  a  student-centered  approach  rather  than  an 
instructor-centered  approach  to  learning  in  DL. 
Student-centered  approaches,  such  as  the  constructiv¬ 
ist  model,  assume  that  students  learn  more  effectively 
if  they  take  responsibility  for  their  learning  and 
discover  information  on  their  own  rather  than  being 
told  what  to  do  by  an  instructor  or  machine.  Indeed, 
Sitzmann  et  al.  (2005)  found  that  in  comparison 
with  students  in  RL,  students  in  Web-based  courses 
acquired  more  declarative  knowledge  when  the  courses 
had  high  levels  of  learner  control  than  did  students  in 
Web-based  courses  with  low  levels  of  learner  control. 
In  comparisons  of  classroom  instruction  with  and 
without  computer-assisted  instruction,  Kulik  (1994) 
found  that  classes  with  assisted  instruction  produced 
positive  effects  on  learning  when  compared  with 
traditional  instruction,  especially  when  the  computer 
acted  as  a  tutor — that  is,  when  it  divided  material  into 
a  sequence  of  small  steps,  sought  learner  responses 
at  each  step,  and  provided  immediate  feedback  after 
each  response.  Because  many  technologies  used  in  DL 
permit  or  require  self-guided  work,  a  model  of  training 
and  the  assessment  of  training  that  focuses  on  what 
students  are  doing  as  they  learn  seems  appropriate. 


Examples  of  research  questions  pertaining  to  self- 
guided  learning  include  the  following: 

•  What  types  of  student-centered  approaches  are 
appropriate  for  different  forms  of  DL  and  what  are 
the  effects  on  learning? 

•  Flow  do  variations  in  course  structure — for  exam¬ 
ple,  number  of  assignments,  frequency  of  dead¬ 
lines,  and  requirements  for  collaboration — affect 
self-paced  learning  and  students’  success  in  DL? 

•  What  student  characteristics  influence  success  in 
self-guided  learning  and  how  should  information 
about  student  individual  differences  affect  devel¬ 
opment  and  delivery  of  DL?  For  instance,  should 
the  degree  of  course  structure  or  learner  control 
afforded  to  students  be  based  on  such  character¬ 
istics  as  locus  of  control  or  mastery  (learning) 
versus  performance  (outcome)  orientations  (e.g., 
Wang  and  Newlin,  2000)? 

Examine  Specific  Technology  Configurations 

A  wide  variety  of  technologies,  including  videotapes, 
computer-based  training,  Web-based  instruction,  and 
VTT,  may  be  used  in  DL.  Technologies  vary  with 
respect  to  such  characteristics  as  whether  the  instruc¬ 
tion  occurs  in  real  time  (e.g.,  VTT)  or  asynchro¬ 
nously  (e.g.,  computer-based  training),  the  depth  of 
communication  they  permit  between  the  instructor 
and  the  student  (e.g.,  voice  versus  text-based  media), 
and  the  degree  of  interactivity  between  the  informa¬ 
tion  source  and  the  student  (e.g.,  watching  a  video¬ 
tape  versus  playing  a  role  in  a  constructive  or  virtual 
simulation).  Examining  the  effectiveness  of  different 
technology  configurations  for  DL  should  provide 
knowledge  that  can  be  used  to  improve  the  design 
and  outcomes  of  Army  training  programs. 

Questions  of  this  sort  include  the  following: 

•  What  technologies  or  combinations  of  technolo¬ 
gies  are  most  effective  for  DL,  and  how  do  these 
technologies  interact  with  characteristics  of  the 
course  topic,  students,  instructors,  and  aspects  of 
the  training  environment? 

•  Flow  can  the  Army  make  the  most  effective  use 
of  advanced  DL  technologies,  such  as  intelligent 
tutoring  and  virtual  and  constructive  simulations? 

Examine  Strategies  for  Facilitating  Interaction  in 
Distributed  Learning 

One  challenge  of  DL  for  learning  is  that  it  restricts 
communication  between  instructors  and  students 
and  among  students.  Studies  of  collaborative  learn¬ 
ing  show  that  interaction  among  students  is  an 
important  source  of  knowledge  acquisition  for  a 
variety  of  learning  tasks  (e.g.,  Slavin,  Hurley,  and 
Chamberlain,  2003).  A  number  of  aspects  of  com- 
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municating  at  a  distance  are  likely  to  interfere  with 
this  process  (Straus  and  Olivera,  2000).  For  instance, 
communicating  via  such  modes  as  telephone,  e-mail, 
and  online  chat  can  require  more  planning  or  effort 
and  are  typically  less  elaborate  or  detailed  than  the 
types  of  interactions  that  occur  when  students  and 
instructors  are  collocated.  These  features  of  technolo¬ 
gies  for  DL  can  create  obstacles  to  achieving  a  shared 
understanding  of  course  material  and  may  disrupt 
opportunities  for  students  to  learn  from  instructors 
and  each  other. 

Examples  of  questions  that  address  communica¬ 
tion  in  DL  include  the  following: 

•  What  aspects  of  communication  (e.g.,  frequency, 
media  used)  contribute  to  better  outcomes? 

•  Under  what  conditions  can  and  should  collabora¬ 
tion  between  students  be  fostered  in  DL,  and 
what  mechanisms  can  be  used  to  support  collabo¬ 
ration? 

•  What  strategies  (e.g.,  peer  tutors)  can  be  used  to 
overcome  some  of  the  problems  of  mentoring  at  a 
distance? 

Examine  incentives,  training,  and  support 
systems  for  distributed  learning  developers 
and  instructors 

DL  instruction  requires  different  methods  of  design¬ 
ing  training,  delivering  material,  and  interacting  with 
students  than  does  traditional  classroom  training.  We 
have  seen  little  research  that  delineates  what  skills  are 
needed  and  how  to  train  instructors  to  teach  in  DL 
environments.  In  addition,  our  interviews  with  train¬ 
ing  staff  suggested  that  current  incentive  schemes  are 
inconsistent  with  the  requirements  for  the  design  and 
delivery  of  effective  DL  courses. 

Examples  of  questions  that  address  the  development 
and  application  of  these  skills  include  the  following: 

•  What  is  the  effect  of  providing  training  to  instruc¬ 
tors  for  teaching  in  DL  environments? 

•  What  compensation  and  incentive  structures  are 
needed  to  motivate  trainers  to  use  appropriate 
instructional  practices  for  DL? 

•  Flow  do  different  incentive  structures  for  organic 
training  developers  and  external  contractors  affect 
efficiency  in  course  development  (e.g.,  through 
sharing  and  reuse  of  training  content)  and  course 
quality? 

Improve  the  Quality  of  Research  on 
Distributed  Learning 

In  addition  to  shifting  the  focus  of  its  research  on 
DL,  the  Army  should  strive  to  improve  the  quality  of 
this  research. 


Use  Random  Assignment  Whenever  Possible  and 
Measure  Relevant  Covariates  of  Learning 

In  studies  that  examine  different  interventions  or  ways 
of  implementing  DL  (for  instance,  different  technol¬ 
ogy  configurations  or  different  types  of  training  for  DL 
instructors),  randomly  assigning  participants  to  condi¬ 
tions  is  the  best  way  to  ensure  that  any  differences  in 
performance  between  the  groups  can  be  reliably  attrib¬ 
uted  to  the  intervention.  When  random  assignment  is 
impractical  (e.g.,  in  field  studies  of  training  in  exist¬ 
ing  groups),  it  is  especially  important  to  measure  and 
control  statistically  for  the  effects  of  variables  that  can 
typically  affect  performance  in  training;  such  variables 
include  student  cognitive  ability,  personality  traits, 
experience  with  course  material,  and  demographic 
characteristics.  In  the  context  of  DL,  variables  such 
as  previous  experience  with  DL  and  computer  self- 
efficacy — one’s  confidence  in  using  computers — may 
also  affect  student  performance. 

Use  Large  Sample  Sizes  or  More  Powerful 
Study  Designs 

Findings  of  “no  difference”  between  DL  and  RL  can¬ 
not  be  reliably  interpreted  to  mean  that  DL  and  RL 
are  equally  effective.  In  the  language  of  research,  one 
cannot  confirm  the  “null  hypothesis.”  To  be  confident 
that  a  finding  of  no  difference  is  meaningful,  one 
needs  sufficient  statistical  power,  which  is  determined, 
in  part,  by  sample  size.5  For  example,  if  one  is  plan¬ 
ning  a  study  that  compares  two  groups  of  students — 
one  that  receives  RL  and  one  that  receives  DL — 393 
students  are  needed  in  each  group  to  conclude  that  an 
absence  of  difference  between  groups  is  meaningful 
(Cohen,  1988). 6  Many  previous  studies  comparing  RL 
and  DL  lack  sufficient  power  to  conclude  that  a  lack 
of  difference  between  conditions  is  “real.” 

Thus,  when  comparing  alternative  interventions, 
such  as  different  technology  configurations  for  DL, 
and  when  anticipating  a  small  effect,  it  is  important 
to  use  sufficiently  large  samples  or  research  designs 
that  provide  adequate  power.  In  contrast,  if  one 
expects  a  moderate  (or  larger)  effect  of  the  interven¬ 
tion,  a  much  smaller  sample  is  required.7  Researchers 
also  can  use  study  designs  that  require  smaller  sam¬ 
ples.  For  instance,  fewer  study  participants  typically 
are  needed  to  determine  whether  a  single  intervention 


5  The  larger  the  sample  size,  all  other  things  being  equal,  the  smaller  the 
error  in  measurement  and  the  greater  the  precision  or  reliability  of  the 
results  (Cohen,  1988).  When  results  are  measured  with  greater  precision, 
there  is  a  higher  probability  of  detecting  the  phenomenon  of  interest. 

6  This  assumes  conventional  standards  of  a  small  effect  size  (d  =  .2),  alpha 
=  .05,  and  power  =  .80  (Cohen,  1988). 

7  For  instance,  if  one  expects  a  moderate  effect  size  (d  =  .5)  with  alpha  = 
.05  and  power  =  .8,  64  students  are  needed  in  each  group. 
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makes  a  difference  (e.g.,  improves  learning)  or  when 
the  same  participants  experience  multiple  interven¬ 
tions  than  when  comparing  alternative  interventions 
among  different  groups  of  participants.8 

Use  a  Multivariate  Approach  to  Measuring 
Training  Success 

The  effectiveness  of  training  can  be  measured  many 
ways.  Using  multiple  measures  can  help  training 
developers  and  instructors  determine  whether  and 
what  trainees  learn  from  training.  It  is  particularly 
important  to  use  objective  measures  of  learning 
(e.g.,  performance  on  skills  tests)  and  to  measure 
knowledge  retention  and  job  performance  after  stu¬ 
dents  complete  training.  In  particular,  few  studies 
have  measured  job  performance  following  training. 
Other  indicators  of  learning  from  training  include 
self-efficacy  (confidence  in  one’s  ability  to  perform  a 
task),  commitment  to  training  goals,  and  motivation 
(Kraiger,  Ford,  and  Salas,  1993). 

Use  Appropriate  Statistical  Tests  to  Analyze 
Empirical  Results 

We  found  that  in  a  number  of  published  studies, 
incorrect  statistical  tests  were  used,  thus  compro¬ 
mising  the  meaning  and  value  of  results.  Bonk  and 
Wisher  (2000)  provide  a  useful  guide  for  e-learning 
research  intended  for  military  training  developers 
and  planners,  instructional  designers,  and  program 
evaluators.  They  describe  instructional  design  issues, 
appropriate  research  methods,  and  suggested  topics 
for  research. 

Policy  Recommendations  to  Support 
Design,  Implementation,  and  Evaluation 
of  Distributed  Learning  in  the  Army 

To  reap  the  benefits  of  an  expanded  research  effort 
on  the  quality  of  DT,  we  recommend  a  number  of 
policies  for  TADLP  and  other  organizations  that 
are  responsible  for  training  in  the  Army.  These  rec¬ 
ommendations  are  based  on  the  interviews  that  we 
conducted  with  training  developers,  instructors,  and 
other  training  staff. 

Use  Empirically  Tested  Measures  of  Learning 
and  Make  Results  Available  Within  the  Training 
Community 

Comprehensive  assessment  should  become  a  rou¬ 
tine  procedure  in  every  DT  course.  We  suggest  that 
TRADOC  identify  empirically  tested  measures  of 


8  See  Cohen  (1988)  for  recommendations  regarding  sample  sizes  needed 
for  different  study  designs. 


learning  and  provide  a  means  for  training  develop¬ 
ers  and  quality  assurance  personnel  to  access  them 
online.  These  measures  can  then  be  used  to  cre¬ 
ate  course  evaluation  forms  that  are  customized  to 
particular  courses  (Straus  et  ah,  2003).  In  addition, 
making  results  of  course  evaluations  available  to  the 
training  community  at  large  will  provide  data  to  sup¬ 
port  DT  efforts  across  proponent  schools. 

Develop  In-House  Expertise  in  the  Development 
and  Assessment  of  Distributed  Learning  Courses 

We  suggest  that  the  Army  develop  in-house  expertise 
in  the  development  and  assessment  of  DL  courses  to 
avoid  becoming  overly  dependent  on  contractors  and 
to  make  more  cost-effective  use  of  their  services.  The 
more  Army  personnel  understand  about  how  to  develop 
and  evaluate  DT  products,  the  better  they  will  be  at 
writing  appropriate  contracts,  conducting  oversight 
of  product  development,  and  providing  appropriate 
quality-assurance  testing.  Key  practices  the  Army 
should  develop  to  become  informed  consumers  are 
described  in  Shanley  et  al.  (2005,  Chapter  3). 

One  way  for  the  Army  to  develop  this  expertise 
is  to  use  public-private  partnerships  in  which  Army 
training  personnel  work  alongside  contractors  and 
eventually  assume  responsibility  for  the  training 
function — an  approach  that  state  governments  have 
found  effective  for  implementing  and  managing 
information  technology  projects  (Anderson  et  al.,  2003). 

Enhance  Opportunities  for  Developers  and 
Trainers  to  Communicate 

Course  development  and  evaluation  are  generally 
decentralized  in  the  proponent  schools.  Our  discus¬ 
sions  with  personnel  at  various  schools  suggest  that 
the  training  development  community  would  benefit 
from  additional  opportunities  for  formal  or  informal 
interaction.  For  instance,  we  found  that  DT  modules 
developed  at  one  school  were  applicable  to  courses 
being  developed  at  other  schools,  but  the  training 
developers  were  not  aware  of  these  potential  syner¬ 
gies.  Tikewise,  useful  information  about  the  effective¬ 
ness  of  DT  may  not  be  routinely  disseminated  in  the 
Army  training  community. 

The  Army  already  offers  several  mechanisms  for 
information-sharing,  including  the  TADLP  Bulletin, 
the  Army  Training  Support  Center  Individual 
Training  Support  Directorate  DL  conferences,  and 
TRADOC  Knowledge  Net  (TKN),  which  includes 
collaborative  tools  such  as  threaded  discussions, 
instant  messaging,  and  repositories  where  staff  can 
post  and  download  documents.  Apparently,  use  of 
TKN  has  been  limited,  perhaps  in  part  because  of 
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a  lack  of  staff  to  market  and  manage  it.  We  recom¬ 
mend  that  TRADOC  provide  the  staff  and  financial 
resources  needed  to  promote  information  exchange 
and  build  communities  of  practice  among  training 
staff.  Further,  it  may  be  helpful  to  establish  an  advi¬ 
sory  group  made  up  of  training  staff  from  the  schools 
to  guide  course  development  and  assessment  efforts. 


Resources  must  be 
available  to  update 
training  to  keep 
pace  with  the 
Army's  rapidly 
changing  operating 
environment. 


Align  Incentives  and  Other  Organizational 
Systems  with  the  Development,  Implementation, 
and  Maintenance  of  High-Quality  Distributed 
Learning 

In  the  past,  resourcing  practices  have  created  dis¬ 
incentives  for  implementing  effective  DL  courses 
and  keeping  course  material  current.  For  example, 
the  training  personnel  we  interviewed  reported  that 
funding  for  the  delivery  of  DL  courses — which  is 
based  on  the  instructor  contact  hours  metric — does 
not  take  into  account  all  the  required  supporting 
activities,  such  as  off-the-platform  interaction  between 
instructors  and  students  (see  also  U.S.  Department 
of  the  Army,  2002).  Recently,  the  method  of  calcu¬ 
lating  instructor  contact  hours  for  DL  courses  has 
been  modified  to  better  reflect  the  amount  of  inter¬ 
action  required  between  instructors  and  students 
(U.S.  Department  of  the  Army,  TRADOC,  2005). 

Resources  also  must  be  available  to  update  train¬ 
ing  to  keep  pace  with  the  Army’s  rapidly  changing 
operating  environment.  DL  courseware  maintenance 
and  improvement  has  been  persistently  underfunded. 
As  noted  in  the  U.S.  Army  Audit  Agency's  (AAA) 
review  of  courseware  development  for  DL,  many 
courses  are  or  will  be  obsolete  before  they  are  imple¬ 
mented  (U.S.  Department  of  the  Army,  2002). 
Creating  and  maintaining  DL  is  essentially  a  soft¬ 
ware  development  process  (Shanley  et  al.,  2005); 
software  life-cycle  and  financial  support  models 
might  inform  TRADOC  on  how  to  resource  DL  for 
the  longer  term.  In  addition,  the  AAA  recommended 
that  the  Army  establish  a  maintenance  strategy  that 
specifies  who  is  responsible  for  updating  courseware 
and  what  procedures  to  follow  if  additional  financial 
resources  are  needed  for  courseware  maintenance. 

To  motivate  training  staff  to  support  DL  efforts, 
performance  evaluation  systems  must  be  aligned  with 
incentives.  Increased  personnel  readiness  and  train¬ 
ing  to  one  standard  are  goals  for  DL  that  might  be 
considered  as  a  basis  for  evaluating  the  performance 


of  training  staff.  However,  we  do  not  recommend 
using  these  outcomes  as  metrics  because  they  are 
typically  beyond  the  control  of  individual  training 
developers  or  instructors.  To  influence  the  behavior 
of  these  training  staff,  incentives  should,  instead, 
be  focused  on  effective  processes  or  behaviors,  such 
as  (1)  applying  instructional  design  principles  in 
DL  course  development  and  delivery,  (2)  creating 
SCORM-compliant  courseware  to  facilitate  reuse  of 
content  among  related  courses,9  (3)  participating  in 
communities  of  practice  to  share  information  about 
DL  efforts,  (4)  conducting  systematic  evaluations  of 
DL  courses  and  revising  courses  based  on  the  results 
of  these  evaluations,  and  (5)  revising  courses  to  keep 
both  the  content  and  the  means  of  delivering  them 
up  to  date.  TRADOC  should  also  develop  metrics  to 
assess  the  broader  goals  of  DL  specified  above  at  the 
proponent  school  level. 

Summary 

Research  on  DL  has  not  kept  pace  with  the  imple¬ 
mentation  of  DL  efforts.  We  recommend  that  the 
Army  continue  to  conduct  research  on  DL,  empha¬ 
sizing  how  to  improve  DL  rather  than  comparing  DL 
and  RL.  Such  research  might  focus,  for  instance,  on 
the  effects  of  student-centered  learning  approaches, 
specific  configurations  of  DL  technologies,  and  strat¬ 
egies  for  facilitating  interaction  among  instructors 
and  students.  Many  other  factors  that  could  contrib¬ 
ute  to  learning  in  DL  could  be  examined  in  this  way. 
We  also  encourage  the  Army  to  improve  the  quality 
of  this  research  by  conducting  studies  that  are  based 
on  more  rigorous  research  designs,  use  larger  sample 
sizes,  and  examine  multiple  aspects  of  learning. 
Finally,  we  recommend  providing  support  for  effec¬ 
tive  DL  development,  implementation,  maintenance, 
and  improvement  through  organizational  resources 
and  personnel  practices.  Such  support  should  include 
establishing  mechanisms  to  make  systematic  evalua¬ 
tion  a  standard  practice  in  DL  courses.  These  recom¬ 
mendations  can  help  the  Army  use  DL  to  realize  its 
vision  of  transforming  training  and  providing  life¬ 
long  learning  for  multiskilled  soldiers.  ■ 


9  SCORM,  or  Shareable  Content  Object  Reference  Model,  refers  to  stan¬ 
dards  that  specify  ways  to  catalog  course  objects  to  enable  interoperability, 
accessibility,  and  reusability  of  learning  content. 
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